evaluating  both  models  were  collected  from  a single  computer  simulation 
program  as  described  in  chapter  I II. A. 3.  Features  of  the  reliability 
testing  procedure  computer  simulation  such  as  phase  planned  test  time 
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PTT.  determination  and  assignment  of  one-half  a failure  in  those  i n- 

i 

stances  when  no  failures  occurred  during  a test  phase  were  in  affect 
for  the  evaluation  of  both  the  failure  rate  models.  Also,  the  same 
twenty-eight  lambda  sets  (failure  rate  sets)  given  in  tables  3.1  and 
3.2  and  the  same  number  of  simulation  replications  ( NS  I MS  = 100)  were 
utilized  for  both  model  evaluations. 

** . Computer  Simulation  Data  Manipulation 

The  continuous  instantaneous  failure  rate  reliability  growth 
model  of  equation  3-37  may  be  written  as 

A.  = (l-a.)b.(TT.)  a*  . (3-3 8) 


As  noted  in  equation  3-16  an  estimate  of  the  unknown,  underlying  instan- 
taneous failure  rate  of  a component  for  any  given  test  phase  of  a reli- 
ability testing  procedure  corrected  for  planned  test  time  bias  is  given 
by 


2NT.  F. 

1 1 

1 + 2NT.  ’ T. 

1 1 


(3.39) 


for  i = 1,  2,  3.  •••  . K where  T.  is  the  total  test  time  accumulated 
for  the  number  of  components  NT.  tested  during  the  phase  and  F.  is 
the  number  of  component  failures  occurring  during  the  phase. 

If  the  logarithmic  transformation  is  applied  to  equation  3.38, 

then 


In  A. 


1 


In  ( 1 -a . ) b . 

1 1 


a. In  TT.  . 
1 1 


(3-^0) 
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To  obtain  the  ordinary  least  squares  regression  estimates  for  a.  and 
b.  the  data  pairs  (In  In  TT j ) are  employed  from  the  results  of  the 

reliability  testing  procedure  computer  simulation.  Let 


Y . - In  \ . , 

i i 


X.  - In  TT.  , 

i i 


Y - i Y Y.  , and 

I I l—J  J 

j-l 

| ' 

x.  - 4-  V*  x. 

1 1 j 

j-i 

for  i - 1,  2,  3.  • • , K.  Then  equation  3 • AO  becomes 


Y.  - In  b . ( 1 -a . ) - a.X. 

i ii  ii 


(3 ■ Al ) 


for  i — I,  2.  3.  ...  . K.  By  the  derivation  presented  in  Appendix  A the 


ordinarv  least  squares  regression  estimates  for  the  instantaneous  failure 
rate  model  parameters  a.  and  b.  are 

i i 


E Vj  • E 1 

j-l  j-l 

I i 

5 1 £\i  ■ E V 

j-'  j-' 


(3-A2) 


exp  ( Y . ♦ a . X . ) 
r i ii 


(3. A3) 


for  i • 2,  3,  A,  , K.  Note  that  as  with  the  cumulative  failure  rate 


model  the  regression  requires  at  a minimum  observations  on  the  results 
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of  two  test  phases;  thus,  model  parameter  estimates  are  made  for  the 
second  thru  the  test  phase.  The  instantaneous  failure  rate  estimate 
given  bv  equation  3-39  was  used  for  the  first  phase  of  testing;  i.e., 


^ 2NT'  F> 
XT)  “ 'l  " F+  2NT j T)  • 


(3- 


As  the  instantaneous  failure  rate  model  parameter  estimates  were 
obtained,  they  were  applied  to  the  model  in  equation  3-38  to  obtain  model 
determined  estimates  for  the  instantaneous  failure  rate  reliability 
status  at  the  end  of  each  phase  of  testing;  i.e., 


VT  - ( 1 - a*.  Vb\  ( TT . ) ' 

T . ill 

i 


(3.45) 


for  i ■ 2,  3.  4,  ...  , K.  Again,  the  simulation  index  r is  left  impl  i 
cit  (r  « I,  2,  3.  •••  . NSIMS  ■ 100). 

Upon  completion  of  all  computer  simulations  (NSIMS  - 100)  for 
each  specified  lambda  set,  performance  statistics  were  computed  as 


T.  NSIMS 


NSIMS 

E ^T. 


l 3 . 46) 


1 NSIMS 

S.D.^  - 1 

f NS  1 MS  - 1 1 V 

, and 

(3.47) 

,r  i 


P.  S.E.<^ 


x 100 


(3.48) 


T.  — 


for  i * I,  2,  3.  . K.  Also,  in  order  to  evaluate  the  forecasting 

performance  of  the  i ns  tan taneous  failure  rate  model  the  same  method  of 
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computing  an  estimate  of  the  total  accumulated  test  time  for  a one  test 
phase  in  the  future  failure  rate  estimate  utilized  for  the  cumulative 
failure  rate  model  in  equation  3-31  was  utilized  for  the  instantaneous 
failure  rate  model  forecasts.  Hence, 


F\ 


i + 1 


where 


(3- 49) 


TT 


i + 1 


TT,  + (NT,  , x PTT.  ,) 
i i+1  i+1 


(3.50) 


for  i - 2,  3,  4 K.  Note  again  that  forecasts  were  made  for  test 

phases  3 thru  K as  was  the  case  with  the  cumulative  failure  rate  mode). 
Forecast  statistics  were  then  computed  as 


NS  IMS 


F\ 


I 


T,  NS  I MS 


E ^T. 


(3.5D 


r-1 


i .r 


S.D. 


Vif 


NS  IMS 


1 


F\_  ▼ NS  I MS-  1 

i r-1 


E <^T. 


I . r 


FV> 

I 


, and 


(3-52) 


S.O. 


FX 


T. 


P.S.E.^ 


x 100 


(3.53) 


T, 


F V 


T, 


for  i - 3,  4,  5,  ...  , K. 

5 . Model  Performance 

a.  Accuracy  Performance 

Figures  3-42  thru  3-77  depict  the  accuracy  performance  of 
the  continuous  i ns tantaneous  failure  rate  reliability  growth  model  for 
selected  tests  per  phase  NT.  cases  of  the  lambda  sets  listed  in  tables 
3-1  and  3-2.  In  each’graph  the  specified  underlying  instantaneous 
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failure  rate  \.  from  table  3-1  or  3-2  ( , solid  line),  the  mean 

model  determined  ins  tan  taneous  failure  rate  from  equation  3-46 

(0  , circles),  and  the  mean  model  forecast  i ns tantaneous  failure  rate 

from  equation  3-51  (X  , crosses)  are  all  plotted  against  the  mean 

i 

total  accumulated  test  time  TT.  from  equation  3.29  for  each  phase  of 
the  specified  reliability  testing  procedure  of  the  acquisition  cycle. 
While  the  specified  underlying  ins tantaneous  failure  rate  is  plotted  as 
a smooth,  continuous  line  for  purposes  of  contrast,  it  should  be  noted 
from  both  figures  3-1  and  3-2  and  the  testing  procedure  design  that  the 
specified  underlying  instantaneous  failure  rate  is  actually  a step  func- 
tion in  shape  with  discontinuities  occurring  at  the  end  of  each  test 
phase . 


Examining  figure  3-47  for  example  at  a mean  total  accumulated 
test  time  of  approximately  205-0  test  time  units  the  specified  instanta- 
neous failure  rate  is  O.jO,  the  mean  model  determined  instantaneous 
failure  rate  is  approximately  0.24,  and  the  mean  model  forecast  instanta- 
neous failure  rate  is  approximately  0.19-  As  noted  on  the  graph  the 
accuracy  performance  plotted  is  for  the  lambda  set  6,  16  test  phase,  20 
tests  per  phase  reliability  testing  procedure  simulation.  Since  the 
point  examined  is  for  the  last  test  phase  (i  • K = 16),  table  3.1  shows 
that  for  lambda  set  6,  test  phase  16  the  specified  instantaneous  failure 
rate  is  0.30  as  expected. 

The  instantaneous  failure  rate  model  frequently  yielded  mean 
forecast  failure  rates  that  were  "of f- the-sca 1 e"  of  the  accuracy  perfor- 
mance graphs;  i.e.,  mean  forecast  failure  rates  greater  than  1.0.  These 
of f- the-sca I e mean  estimates  ranged  from  failure  rates  onlv  slightlv 
greater  than  1.0  to  mean  estimated  failure  rates  of  magnitudes  in  excess 
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of  10  . The  extreme  values  occurred  only  for  the  first  forecast  and 
occasionally  fur  the  second  forecast  (test  phases  3 and  4).  Regardless 
of  the  magnitude  of  these  of f- the-scale  forecast  failure  rates,  all 
points  of  magnitude  greater  than  1.0  were  plotted  at  1.0. 

Figures  3.42  thru  3-46  and  3 - 60  thru  3-64  display  the  instan- 
taneous failure  rate  model's  performance  for  the  "nice"  underlying  fail- 
ure rate  progress  paths.  The  model,  while  discerning  the  shape  of 
these  "nice"  underlying  failure  rate  patterns,  is  consistently  optimistic 
in  both  determining  the  current  failure  rate  status  and  forecasting  the 
next  test  phase  failure  rate,  especially  in  the  case  of  the  longer  six- 
teen phase  reliability  testing  procedures.  The  model  performs  more 
accurately  for  the  more  volatile  six  test  phase  procedures  (figures  3-60 
thru  3-64)  than  for  the  protracted  sixteen  test  phase  procedures  (figures 
3-42  thru  3-46).  In  fact  mean  determined  failure  rate  performance  is 
excellent  for  the  shorter  six  test  phase  cases.  Forecasting  accuracy 
typically  is  of f-the-sca le  for  the  first  test  phase  improving  rapidlv 
by  the  third  or  fourth  test  phase.  Once  stabilized,  the  mean  forecast 
failure  race  generally  is  biased  optimistically  as  expected  from  the 
method  utilized  to  produce  forecasts  (equations  3.49  and  3 - 50 ) . 

In  figures  3.47,  3-48,  and  3.65  thru  3-67  accuracy  of  the 
instantaneous  failure  rate  model  is  displayed  for  essentially  linear 
underlying  failure  rate  progress  paths.  Again,  mean  determined  failure 
rate  performance  shows  that  the  model  detects  the  linear  shape  of  the 
progress  path  with  reasonable  accuracy  but  provides  consistently  opti- 
mistic estimates  of  the  underlying  instantaneous  failure  rate.  Fotecast- 
ing  accuracy,  while  consistent  in  the  sixteen  test  phase  procedures,  is 
very  poor  being  only  around  50%-60$  of  the  magnitude  of  the  underlying 
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i ns  tan taneous  failure  rate  for  the  majority  of  the  test  phases  until 
the  final  one  or  two  phases.  For  the  six  phase  procedures  forecasting 
performance  is  extremely  erratic  as  can  be  seen  in  figures  3-65  thru 
3.67.  In  fact  forecasting  performance  actually  degrades  as  more  test 
information  becomes  available  in  the  case  of  lambda  set  M0D6;  i.e.,  as 
the  number  of  tests  per  phase  increases  from  five  to  twenty  (NT.  » 5 
vs.  NT.  =»  20,  figure  3-65  vs.  3-66).  Also,  figure  3.67  is  for  a twenty 
tests  per  phase  (NT.  * 20)  procedure;  and  therefore,  in  keeping  with  the 
convention  established  for  presenting  accuracy  performance,  graphs  this 
figure  shows  the  best  performance  of  the  instantaneous  failure  rate 
model  for  lambda  set  M0D7-  The  five  and  ten  tests  per  phase  performance 
for  lambda  set  M0D7  were  worse  than  the  performance  shown  in  figure  3-67. 

Figures  3-^9  thru  3-52  and  3-69  thru  3.17  portrav  the  in- 
stantaneous failure  rate  model's  accuracy  performance  for  permanently 
stagnated  reliability  status  cases.  The  most  vivid  contrast  for  these 
cases  is  between  situations  in  which  failure  rate  stagnates  at  a high 
value  (\.  » 0.70;  figures  3.^9,  3-68,  and  3-69)  versus  stagnation  at  a 
low  rate  (\.  = 0.05;  figures  3-52  and  3-71).  When  the  underlying  failure 
rate  stagnates  at  a high  value,  the  instantaneous  model's  mean  determined 
failure  rate  estimates  are  very  erratic  (\.  ■ 0.70;  figures  3-i*9  and 
3.68)  and  the  mean  forecast  estimates  are  virtually  useless  from  a magni- 
tude consideration  being,  even  once  stabilized,  approximately  A0?-50% 
of  the  magnitude  of  the  true  underlying  instantaneous  failure  rates. 

Note  that  figure  3-^*9  is  for  a "best  case"  situation  of  NT.  * 20.  Fore- 
casts at  best  give  an  indication  of  stagnation  when  the  stagnation  occurs 
at  a high  true  underlying  value.  In  the  case  of  lambda  set  M0D8  deter- 
mined estimated  failure  rate  improves  significantly  when  more  testing 
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is  performed;  i.e.,  figure  3-69  versus  figure  3-68  (NT.  ■ 20  vs.  NT.  - 5). 
On  the  other  hand,  when  the  true  underlying  failure  rate  stagnates  at  a 
low  point,  both  the  model's  determined  and  forecast  accuracy  performance 
appear  to  be  excellent  (\.  ■ 0.05,  figures  3-52  and  3.71,  NT,  ■ 5). 

Figures  3-50,  3-51,  and  3-70  show  for  an  intermediate  stagnation  level 
(\.  * 0.30)  how  determined  and  forecast  failure  rate  performance  of  the 
instantaneous  model  transitions  from  poor  to  excellent. 

The  contrast  between  the  underlying  failure  rate  permanent 
stagnation  cases  points  out  a general  performance  character i st i c of  the 
instantaneous  failure  rate  model.  Because  the  model  produces  consistent 
optimistically  biased  determined  and  forecast  failure  rate  estimates,  as 
the  true  underlying  failure  rate  progress  path  approaches  low  failure 
rates  the  model's  accuracy  performance  improves  significantly  by  ..eces- 
s i ty  since  the  estimates  become  "sandwiched"  between  the  low  true  under- 
lying failure  rate  and  0.0.  This  "closing  of  the  brackets"  effect  may 
also  produce  favorable  variability  performance  for  the  i ns  tan taneous 
fa i lure  rate  model . 

Figures  3-53.  3-5^,  3.72,  and  3.73  present  the  accuracy  per- 
formance of  the  instantaneous  failure  rate  model  in  the  case  of  true 
reliability  status  progress  interrupted  bv  a period  of  stagnation.  Only 
"best  case"  (NT,  » 20)  graphs  are  shown.  The  model's  mean  determined 
instantaneous  failure  rate  performance  is  fair  for  these  cases  and  con- 
tinues to  exhibit  consistent  optimistic  bias.  Mean  forecast  failure 
rates  general !v  are  indicative  of  the  underlying  failure  rate  path  trend 
only  and  are  significantly  off  the  mark  in  forecasting  instantaneous 
failure  rate  magnitude.  The  "kick-down"  in  the  mean  forecast  of  the 
failure  rate  for  the  first  phase  after  the  period  of  stagnation  in  figure 
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3.53  (test  phase  II  at  approximately  130.0  total  accumulated  test  time 
units)  is  more  pronounced  in  the  five  and  ten  tests  per  phase  (NT.  ■ 5, 

10)  cases  for  lambda  sets  11  and  12. 

For  situ-'*  ions  of  temporary  reliability  status  degradation 
figures  3.55  thru  3-59  and  3-74  thru  3.77  reveal  that  in  these  "worst 
situation"  cases  the  i ns  tan taneous  failure  model  performs  surprisingly 
well  in  discerning  the  shape  of  the  true  underlying  failure  rate  path, 
especially  for  the  twenty  tests  per  phase  procedures  (NT.  » 20;  figures 
3-57,  3-59,  3.75,  and  3-77).  During  the  period  of  failure  rate  degrada- 
tion, the  instantaneous  model  tends  to  produce  determined  failure  rate 
estimates  that  overstate  the  degree  of  degradation.  This  is  especially 
true  in  the  "least  data"  cases  (NT.  ■ 5,  figures  3-55  and  3 • SS ) of  the 
sixteen  phase  testing  procedures.  This  charac te r i s t i c is  not  as  pro- 
nounced for  the  six  test  phase  procedures  (NT.  ■ 5,  figures  3.74  and 
3-76).  The  model's  mean  forecast  instantaneous  failure  rate  estimates 
remain  optimistic  throughout  the  periods  of  degradation,  but  reflect  the 
shape  of  the  true  underlying  failure  rate  progress  path  with  verv  credible 
fidelity.  The  figure  series  3-55  thru  3-57  indicates  how  mean  determined 
failure  rate  estimating  performance  improves  with  additional  testing 
(NT.  ■ 5,  10,  and  20)  while  mean  forecast  failure  rate  estimating  perfor- 
mance is  very  stable  after  the  third  forecast  for  all  test  per  phase 
sizes.  Again,  note  in  figures  3 - 59  and  3-77  that  when  the  true  instanta- 
neous failure  rate  ha-,  decreased  to  a small  magnitude  (test  phase  16), 
the  instantaneous  model’s  determined  and  forecast  failure  rate  accuracy 
displays  significant  improvement. 

b.  Variability  ^Precision)  Performance 

The  variability  performance  of  the  continuous  instantaneous 
failure  rate  model  is  presented  in  tables  3<!5  thru  3.26.  Entries  in 
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ED/MEAN  INSTANTANEOUS 


. i srEClMEO  i ns r an raucous  failure  rate 
O I mean  mOOEL  DETERMINED  INSTANTANEOUS  FAILURE  rate 
x • MEAN  model  forecast  instantaneous  failure  RATE 


50.00  75.00  100.00  125.00  150.00 

TOTAL  ACCUMULATED  TEST  TIME 


» 75.00 


FIGURE  3.42 

instantaneous  reliability  growth  model  performance 
LAMBOA  set  l:  16  PHASES.  5 TESTS/PHASE 


SPECIFIED/MEAN  INSTANTANEOUS  FAILURE  RATE 


s^fcifieo  instantaneous  failure  rate 

MtBN  MODEL  DETERMINED  I NS TAN TUNE  (JUS  FAILURE  RATE 
MEAN  MOOEl  FORECAST  INSTANTANEOUS  FAILURE  RATE 


70.00  105.00  UO.OO  P5.0Q 

TOTAL  ACCUMULATED  TEST  TIME 


FIGURE  3.43 

INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  0:  16  PHASES.  20  TEST3.,pHASE 


SPEC  I F I EO/MEON  INSTANTANEOUS  FAILURE  RATE 


SPEC  I F I EO/HEAN  INSTANTANEOUS  FAILURE  RATE 


. i SPECIFIED  INSTANTANEOUS  FAILURE  RATE 
o . MEAN  NOOEL  DETERMINED  INSTANTANEOUS  FAILURE  rate 
x I mean  mooel  forecast  instantaneous  failure  RATE 


30.00  60.00  90.00  120.00  150.00 

TOTAL  ACCUMULATED  TEST  TIME 


180.00  210.00 


FIGURE  3.45 

INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMBDA  SET  4:  16  PHASES.  5 TESTS/PHASE 


SPEC  I F IED/MEAN  I NST  ANTANEOUS  FAILURE  RATE 


i SPECIFIED  INSTANTANEOUS  failure  RATE 
» KEAN  MOOEL  determined  INSTANTANEOUS  failure  rate 
. mean  model  forecast  instantaneous  failure  rate 


UO. 00  00.00  120.00  160.00  200.00 

TOTAL  ACCUMULATED  TEST  TIME 


2U0.00  290.00 


FIGURE  3.46 

INSTANTANEOUS  AELIABILITT  GROWTH  MODEL  PERFORMANCE 
LAM80A  SET  5;  16  PHASES.  5 TESTS/PHASE 


[ED/MEfi N INSTANTANEOUS  FAILURE  RATE 


SPEC  I F I EO/MEAN  INSTANTANEOUS  FAILURE  RATE 


SPECIFIED/MEAN  INSTANTANEOUS  FAILURE  RATE 


SPEC  IF IED/HEAN  I NSTANTANEOUS  FAILURE  RATE 


EO/MEAN  INSTANTANEOUS 


• SPECIFIED  INSTANTANEOUS  FAILURE  RATE 
i MEAN  HOOEL  OETCRHINEO  I NSTANT ANCOUS  FAILURE  RATE 
i HEAN  HOOEL  FORECAST  INSTANTANEOUS  FAILURE  RATE 


90.00  135.00  180.00  225.00 

TOTAL  ACCUMULATED  TEST  TIME 


FIGURE  3.51 

INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBOA  SET  9;  16  PHASES,  20  TESTS/PHASE 


SPEC  I F 1 ED/MEAN  INSTANTANEOUS  FAILURE  RATE 


. « srecifieo  instantaneous  failure  rate 

o I MEAN  MODEL  DETERMINED  INSTANTANEOUS  FAILURE  RATE 
x « hean  MOOEL  FORECAST  INSTANTANEOUS  failure  rate 


9 9 


65.00  130.00  195.00  260.00  325.00 

TOTAL  ACCUMULATED  TEST  TIME 


9 9 


390.00  U55.00 


FIGURE  3.52 


INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMBDA  SET  10:  16  PHASES.  5 TESTS/PHASE 


SPEC  I F I EO/MEAN  1 NST  ANT  ANE0US  FAILURE  RATE 


. i specified  instantaneous  failure  rate 

O i MEAN  HOD  CL  DETERMINED  J NS  TANTANE  OUS  FAILURE  RATE 
X i MEAN  MOOEL  FORECAST  INSTANTANEOUS  FAILURE  RATE 


® © © 


O ® © © © 


XXX 


0 35.00  70.00  105.00  1U0.00  175.00  210.00  2U5.00 

TOTAL  ACCUMULATED  TEST  TIME 


FIGURE  3.53 


instantaneous  reliabilitt  growth  model  performance 

LAM80A  SET  11:  16  PHASES.  20  TESTS/PHASE 


SPECIF IEO/MEAN  INSTANTANEOUS  FAILURE  RATE 


SPEC  I F I ED/ME AN  INSTANTANEOUS  FAILURE  RATE 


SPEC  IF IEO/MEAN  INSTANTANEOUS  FAILURE  RATE 


. i specified  instantaneous  failure  rate 

o i MEAN  HOOEL  DETERMINED  INSTANTANEOUS  FAILURE  RATE 
X i MEAN  HOOEL  FORECAST  INSTANTANEOUS  FAILURE  RATE 


X x 


0 20.00  40.00  60.00  80.00  100.00  120.00  140.00 

TOTAL  ACCUMULATED  TEST  TIME 


FIGURE  3.56 

instantaneous  reliabilitt  growth  model  performance 

LAMBDA  SET  13:  16  PHASES,  10  TESTS/PHASE 


SPEC  I F I ED/MEAN  INSTANTANEOUS  FAILURE  RATE 


i specified  iKsroNTANecus  ibuure  rbtf 
. HE BN  HOOEL  OETERhINEO  I NST BN TBnEOUS  FBIluRE  RKTt 
i HERN  HOOEL  POBECBST  i NSTBNfBNEOUS  FBI  LURE  ROTE 


35.00  70.00  105.00  U0.00  P5.00 

total  accumulated  test  time 


2 1 0. 00 


2*45. 00 


FIGURE  3. 57 


INSTANTANEOUS  AELIABILITT  GROWTH  MODEL  PERFORMANCE 
LAMBOA  SET  13:  16  PHASES.  20  TESTS/PHASE 


SPEC  IF IEQ/MEAN  INSTANTANEOUS  FAILURE  RATE 


SPEC  I F I EO/HERN  INSTANTANEOUS  FAILURE  RATE 


- i SPECIFIED  INSTANTANEOUS  FAILURE  RATE 
Q i mean  hooel  determined  instantaneous  FAILURE  RATE 
x • mean  model  forecast  INSTANTANEOUS  failure  rate 


110.00  165.00  020.00  275.00  330.00  385.00 

TOTAL  ACCUMULATED  TEST  TIME 


FIGURE  3.59 

INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBDA  SET  14;  16  PHASES.  20  TESTS/PHASE 


EO/MEAN  INSTANTANEOUS 


SPEC  I F I ED/MEAN  INSTANTANEOUS  FAILURE  RATE 


I specified  I nsta* raucous  failure  bate 

« MEAN  HOOCL  OCTE  BUNCO  I MS  TAN  tone  airs  EAlLOflE  BATE 
, beam  BODE!.  FORECAST  1 NS  TON  TPn  EDITS  FAILURE  BATE 


iiToo  50.00  7^.00  90.00 

TOTAL  ACCUMULATED  test  TIME 


:05. 00 


FIGURE  3.62 

INSTANTANEOUS  RELIABILITY  growth  model  PERFORMANCE 
LAM0OA  SET  M003:  6 PHASES.  5 TESTS/PHPSE 


SPEC  I F I EO/HEfiN  INSTANTANEOUS  FAILURE  RATE 


srcciFiEO  IN5TBNTflNE0U3  failure  rate 

mean  model  DETERMINED  instantaneous  FAILURE  rate 

MEAN  MODEL  FORECAST  INSTANTANEOUS  FAILURE  RATE 


total  accumulated  TEST  TIME 


FIGURE  3.63 

INSTANTANEOUS  REL  I A0 I L I T T GROWTH  MOOEL  PERFORMANCE 
LAMBDA  SET  MOON:  6 PHASES.  5 TESTS/PHASE 


SPEC  I F I EO/MERN  INSTANTANEOUS  FAILURE  RATE 


SPECIF IEO/HEAN  INSTANTANEOUS  FAILURE  RATE 


EO/MEAN  INSTANTANEOUS  FAILURE  RATE 


SPEC  IF IED/MEAN  INSTANTANEOUS  FAILURE  RATE 


5PEC I F IED/MEAN  INSTANTANEOUS  FAILURE  RATE 


. i SPEC  in  CD  INSTANTANEOUS  failure  rate 
q • mean  MODEL  DETERMINED  instantaneous  failure  rate 
x I MEAN  model  forecast  instantaneous  failure  rate 


<4.00  6.00  3.00  10.00 

TOTAL  ACCUMULATED  TEST  TIME 


12.00 


l <4. 00 


FIGURE  3.68 

INSTANTANEOUS  RELIABILITY  GROWTH  MODEL  PERFORMANCE 
LAMBOA  SET  M0D8;  6 PHASES.  5 TESTS/PHASE 


31 


SPEC  IF IEO/MEAN  INSTANTANEOUS  FAILU 

0.00  O.tO  0.20  0.30  O.UO  0.50  0.60 

I I I I I I 


. i SRECIFIEO  INSTANTANEOUS  FAILURE  BATE 
o • mean  MOOEL  OETERHINED  instantaneous  failure  rate 

x i MEAN  MOOEL  FORECAST  INSTANTANEOUS  FAILURE  RATE 


16.00  24.00  32.00  40.00 

TOTAL  ACCUMULATED  TEST  TIME 


48.00 


56.00 


FIGURE  3.69 

INSTANTANEOUS  P IABILITT  GROWTH  MODEL  PERFORMANCE 
LAMBOA  SET  M „08 : 6 PHASES,  20  TESTS/PHASE 


I 


ED/MEAN  INSTANTANEOUS 


SPECIFIED  INSTANTANEOUS  FAILURE  RATE 

MEAN  MODEL  DETERMINED  instantaneous  failure  rate 

mean  model  forecast  instantaneous  FAILURE  rate 


total  accumulated  test  time 


FIGURE  3.70 

instantaneous  AELIABILITt  growth  model  performance 
LAM80A  SET  M009:  6 PHASES.  5 TESTS/PHASE 


SPEC  I F IEO/MEAN  INSTANTANEOUS  FAILURE  RATE 


. I SPECIFIED  INSTANTANEOUS  FAILURE  RATE 
© i NEAN  NOOEL  DEtERH|nED  instantaneous  FAILURE  rate 
x I NEAN  NOOEL  FORECAST  INSTANTANEOUS  FAILURE  RATE 


25.00  50.00  75.00  100.00  125.00 

TOTAL  ACCUMULATED  TEST  TIME 


150.00  I75.no 


FIGURE  3.71 

instantaneous  reliability  growth  model  performance 

LAMBDA  SET  M0010:  6 PHASES.  5 TESTS/PHASE 


SPECIF IEO/MEAN  1 NST  ANT  ANEQUS  FAILURE  RATE 


ED/MEAN  INSTANTANEOUS 


x 


. i specified  instantaneous  failure  rate 
O i mean  mooel  determined  INSTANTANEOUS  failure  rate 
x i mean  mooel  forecast  INSTANTANEOUS  failure  rate 


30.00  60.00  90.00  120.00  150.00 

TOTAL  ACCUMULATED  TEST  TIME 


180.00  210.00 


FIGURE  3.73 

instantaneous  reliability  growth  model  performance 

LAMBOA  SET  M0D12:  6 PHASES,  20  TESTS/PHASE 


I EO/MEAN  INSTANTANEOUS  FAILURE  RATE 


SPECIFIED/MEAN  INSTANTANEOUS 


ED/MEAN  INSTANTANEOUS  FAILURE  RATE 
30  0.40  0.50  0.60  0.70 


SPEC  IF IED/MEAN  INSTANTANEOUS  FAILURE  RATE 


. i SPECIFIED  I NSTANTANEOUS  FAILURE  RATE 
o I mean  hooel  DETERMINED  INSTANTANEOUS  failure  rate 
x I MEAN  mooel  forecast  instantaneous  failure  rate 


30.00  60.00  90.00  120.00  150.00 

TOTAL  ACCUMULATED  TEST  TIME 


190.00 


210.00 


FIGURE  3.77 

INSTANTANEOUS  RELIABILITY  GROWTH  MOOEL  PERFORMANCE 
LAMBOA  SET  MQ014:  6 PHASES.  20  TESTS/PHASE 


these  tables  are  the  percentage  standard  errors  as  computed  in  equations 
3.^8  and  3-53.  From  figure  3-1*7  the  mean  model  determined  instantaneous 
failure  rate  was  read  as  0.2l*  for  the  sixteenth  test  phase  of  the  lambda 
set  6,  16  test  phase,  20  tests  per  phase  reliability  testing  procedure. 

In  table  3-17  the  percentage  standard  error  corresponding  to  the  sixteenth 
test  phase  of  the  lambda  set  6,  NT.  = 20  simulations  is  2l*% - 8y  equation 
3.48  the  standard  deviation  of  the  i ns  tan taneous  failure  rate  model's 
determined  ins  tan taneous  failure  rate  for  this  phase  is  then 


S 


D. 


P.S.E.  \ 

' 16 


100 


24  x 0.24 
100 


0.0576  . 


(3.55) 


A quick  examination  of  tables  3-15  thru  3-26  reveals  that  the 
goal  of  30%  percentage  standard  error  of  failure  rate  estimates  utilized 
in  evaluating  the  variability  performance  of  the  cumulative  failure  rate 
reliability  growth  model  (chapter  III.A.S.b.)  will  be  too  restrictive  for 
evaluating  the  instantaneous  failure  rate  model's  variability  performance 
Also,  unlike  the  variability  performance  of  the  cumulative  failure  rate 
model  which  almost  uniformly  improved  as  test  phases  of  the  testing  pro- 
cedure were  completed,  variability  performance  of  the  instantaneous  model 
oscillates  for  many  lambda  sets.  For  example,  keeping  in  mind  that  model 
variability  performance  begins  with  the  percentage  standard  errors  re- 
corded for  phase  2,  the  instantaneous  model's  variability  performance  for 
lambda  set  1,  five  tests  per  phase,  determined  failure  rate  in  table  3-15 
improves  steadily  from  76%  in  phase  2 to  38%  in  phase  11.  In  phase  12 
the  model's  variability  performance  jumps  up  to  41%,  improves  back  to 
32%  by  phase  14,  and  finally,  climbs  to  35%  for  the  last  test  phase, 
phase  16. 
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This  oscillation  in  the  variability  performance  of  the  in- 
stantaneous failure  rate  model  means  that  even  if  a reasonable  percent- 
age standard  error  benchmark  is  selected,  determining  the  first  phase 
for  which  variability  performance  on  all  lambda  sets  is  within  the  goal 
does  not  guarantee  the  performance  goal  will  be  satisfied  for  all  subse- 
quent test  phases  of  the  acquisition  cycle. 

Tables  3-15,  3-16,  and  3-17  display  the  instantaneous  failure 
rate  reliability  growth  model's  variability  performance  for  determined 
instantaneous  failure  rate  estimates  for  the  sixteen  test  phase  proce- 
dures. If  a percentage  standard  error  goal  of  40%  is  set,  the  goal  is 
never  achieved  for  NT.  = 5 or  NT.  = 10  (tables  3-15  and  3-16),  but  is 
achieved  for  NT.  = 20  (table  3-17)  on  test  phases  9 7,  8,  14,  and  16. 
Again,  the  oscillating  character  of  the  instantaneous  model's  variability 
performance  is  displayed  by  this  method  of  examination.  Tables  3-18, 
3.19,  and  3-20  show  that  in  the  case  of  the  contracted  six  test  phase 
procedures  the  instantaneous  failure  rate  model's  variability  performance 
for  determined  instantaneous  failure  rate  fails  to  attain  the  h0%  goal 
for  all  simulated  tests  per  phase  specifications;  i.e.,  NT.  = 5,  10,  and 
20. 

In  both  the  sixteen  and  six  test  phase  reliability  testing 
procedures  determined  failure  rate  estimates  given  by  the  instantaneous 
model  were  least  accurate  for  lambda  set  8 and  M008.  See  figures  3-49, 
3-68,  and  3-69.  Also,  the  model's  accuracy  performance  for  these  lambda 
sets  was  the  only  case  in  which  no  definite  bias  trend  was  evident.  The 
instantaneous  failure  rate  model's  variability  performance  for  determined 
ins tantaneous  failure  rate  estimates  is  almost  uniformly  the  worst  for 
lambda  sets  8 and  M0D8  as  evidenced  in  tables  3-15  thru  3-20.  This  was 
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to  be  expected  based  on  the  model's  accuracy  performance  for  these  par- 
ticular specified  underlying  instantaneous  failure  rate  paths. 

Tables  3-21  thru  3-26  contain  variability  performance  results 
of  the  instantaneous  model's  forecast  failure  rates  for  both  the  sixteen 
and  six  test  phase  reliability  testing  procedures.  Percentage  standard 
error  entries  of  1000%  indicate  that  the  actual  percentage  standard 
errors  in  these  cases  were  greater  than  or  equal  to  1000%.  As  with  the 
cumulative  failure  rate  model,  lambda  set  7,  12,  14,  M0D7,  M0D12,  and 
MOD  1 4 provide  difficulty  for  the  instantaneous  model.  These  lambda  sets 
are  the  ones  which  take  the  most  rapid  "plunge"  to  a low  specified  instan- 
taneous failure  rate  of  X^  = 0.0500  or  X ^ = 0.0500.  Again,  if  these 
lambda  sets  are  not  considered,  then  a 40%  percentage  standard  error  goal 
is  achieved  on  phase  14  for  NT.  * 5 and  10  and  on  phase  11  for  NT.  =*  20 
in  the  case  of  the  sixteen  phase  reliability  testing  procedure  simula- 
tions. The  instantaneous  model's  variability  performance  for  forecast- 
ing during  six  test  phase  cycles  is  very  poor  achieving  the  40%  goal  on 
the  last  test  phase  of  the  "most  data"  case  (NT.  * 20)  for  only  seven  of 
the  fourteen  lambda  sets. 


C.  DISCRETE  RELIABILITY  GROWTH  MODEL 


1 . Model  Description 

The  discrete  reliability  growth  model  evaluated  is  of  the  form 


R 

m 


1 

exp  (oi  + Sm) 


(3-56) 


where  R * component  reliability  after  the  m mod i f i cat  ion  of  the 
component 

m = number  of  modifications  that  have  been  made  to  the  component, 
and 

a, 6 ■ constants  that  determine  the  change  in  the  model  estimated 
component  reliability  as  modifications  to  the  component  are 
accompl i shed. 


i 
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INSTANTANEOUS  RELIABILITY  GROWTH  MOCEL  VARIABILITY  PERFORMANCE 


IfvST  ANTANEOUS  RELIABILITY  GROWTH  KCDEL  VARIABILITY  PERFORMANCE 
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The  modification  number  m takes  on  values  0,  1,2,  . . . with  m » 0 
designating  the  original  version  of  the  component.  Component  reliability 
R^  may  be  thought  of  as  an  instantaneous  or  modification  reliability  as 
opposed  to  a cumulative  or  average  reliability  since  it  is  the  intrinsic 
reliability  of  the  mth  modification  version  of  the  item  rather  than  a 
characteristic  reliability  for  all  m+1  versions  of  the  item  that  have 
been  produced.  Under  the  assumption  of  reliability  growth,  the  quantity 
exp  (a  + dm)  in  equation  3-56  is  expected  to  increase  as  modifications 
to  the  component  are  made  which  hopefully  improve  its  reliability.  Hence, 
as  modifications  are  accomplished  the  fraction  1/exp  (a  + 6m)  in  equation 
3-56  decreases  and  component  reliability  increases  toward  the  maximum 
reliability  of  1.0.  The  model  is  considered  discrete  since  the  single 
independent  or  control  variable  m,  modification  number,  is  discrete  as 
opposed  to  a continuous  variable  such  as  total  accumulated  test  time. 

A mathematical  analysis  of  the  discrete  reliability  growth  model 
is  given  in  reference  1.  Other  Monte  Carlo  simulation  evaluations  and 
comparisons  of  the  discrete  model  are  contained  in  references  5 and  9. 


2 . Reliability  Testing  Procedure 


is  modified  again  is  denoted  by  NTF^  • The  total  number  of  item  fail- 


ures NTF^  ' s specified  before  reliability  testing  is  conducted  and  may 
be  varied  from  modification  to  modification.  During  testing  of  each 
modification  version  of  an  item,  an  underlying,  inherent  reliabilitv 
that  is  unknown  to  the  project  managers/contractors  is  present  in  the 


item.  The  underlying  modification  reliability  of  the  m modification 
version  of  the  component  is  denoted  as  R^  . 

Between  each  failure  of  the  modification  version  of  an  item 
a number  of  tests  of  the  item  are  conducted  which  are  denoted  as  NT 

m,  j 

i.e.,  the  number  of  tests  run  on  the  m^  modification  version  of  the 

item  between  the  (j-1)  failure  and  the  j failure.  NT  . includes 

m,j 

the  j failure.  So,  NT^  ^ is  the  number  of  tests  performed  thru  the 
first  observed  failure  of  the  item  when  the  underlying  modification  reli- 
ability is  R . NT  is  the  number  of  tests  performed  from  the  first 
m m,2  r 

failure  thru  the  second  failure  of  the  item  while  the  underlying  modifi- 


cation reliability  remains  R and  so  on.  After  NTF  failures  of  the 

m m 

item  under  test  are  observed,  a modification  is  accomplished  and  the 

underlying  modification  reliabilitv  of  the  component  changes  to  • 

Therefore,  the  failure  number  index  j runs  from  1 to  NTF  for  each 

J m 

modification  m ; i.e.,  NT  , , NT  ,,...,  NT  All  NT  . are 

m, 1 m,2  m.NTF  m, j 

m 

random  variables  and  cannot  be  specified  beforehand.  This  is  quite  dif- 
ferent from  the  testing  procedure  for  the  continuous  reliability  growth 
models  evaluated  where  the  number  of  tests  per  test  phase  were  specified 
prior  to  any  testing  being  performed. 

Figure  3-78  is  a schematic  diagram  of  the  discrete  model  rcli- 
ab  i 1 i ty  testing  procedure.  The  total  number  of  modifications  NTH  and 

the  total  number  of  failures  permitted  before  modification  NTF  are 

m 


fc 
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SCRETE  RELIABILITY  GROWTH  MODEL  RELIABILITY  TESTING  PROCEDURE  DIAGRAM 
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specified  prior  to  testing.  The  number  of  tests  performed  between  fail- 
ures NT  . for  each  modification  version  of  the  item  under  test  consti- 
m,j 

tute  the  data  collected  from  the  reliability  testing  procedure.  As  with 

the  continuous  reliability  growth  models  the  most  salient  feature  of  the 

computer  simulation  of  the  discrete  model  reliability  testing  procedure 

is  that  the  analyst  also  specifies  the  underlying  modification  reliability 

R that  is  effective  for  each  modification  version  of  the  item.  Thus  an 
m 

accurate  assessment  of  the  accuracy  of  the  model  can  be  made  because  the 

R values  are  known, 
m 

3 . Reliability  Testing  Procedure  Computer  Simulation 
a.  Summary 

The  reliability  testing  procedure  appropriate  to  the  discrete 
reliability  growth  model  was  simulated  on  the  Naval  Postgraduate  School 
W.  R.  Church  Computer  Center's  IBM  360/67  System  utilizing  standard  com- 
puter simulation  techniques.  Reliability  testing  procedure  design  para- 
meters NTM  and  NTF  were  specified  alonq  with  various  underlying  mod i - 

m 

fication  reliability  sets.  Tests  of  components  were  then  simulated  with 

component  success  or  failure  being  determined  for  each  test  based  on  the 

appropriate  underlying  modification  reliability  R . Number  of  tests 

m 

between  failures  NT  . data  were  collected  as  each  orocedure  simulation 

m,j 

was  accomplished.  Simulations  of  reliability  testing  procedures  for  each 
specified  modification  reliability  set  were  replicated  one-hundred  times 
in  order  to  assess  the  discrete  reliability  growth  model's  variability 
performance . 

After  the  reliability  testing  procedure  simulations  were  com- 
pleted, discrete  reliability  growth  model  estimates  of  the  underlying  modi- 
fication reliability  of  the  component  were  produced  based  on  the  simulated 
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1 

test  data  by  computations  utilizing  ordinary  least  squares  regression 
techniques  to  estimate  the  model  parameters  ot  and  6 . Mean  and 
variability  statistics  of  the  model  reliability  estimates  were  then  com- 
puted to  permit  performance  evaluation  of  the  discrete  reliability 
growth  model . 

b.  Detail  Description 

The  computer  simulation  of  the  reliability  testing  procedure 
appropriate  to  the  discrete  reliability  growth  model  was  initiated  by 
reading  in  the  following  reliability  testing  procedure  design  specifi- 
cations: NTM,  the  total  number  of  modifications  to  be  made  for  the 

acquisition  cycle  under  consideration;  NTF  , the  total  number  of  failures 

m 

of  the  mC^  modification  version  of  the  item  to  be  observed  until  the 
£ h 

(m+1)  modification  is  made;  R , the  underlying,  intrinsic  modification 

m 

t h 

reliability  of  the  m modification  version  of  the  item;  and  NSIMS,  the 
number  of  times  the  specified  reliability  testing  procedure  was  to  be 
simulated.  For  this  thesis  NSIMS  was  always  specified  as  one-hundred 
simulations. 

After  specification  data  were  read  into  the  computer,  uniform 

(0,1)  random  variates  were  drawn  in  sequence  corresponding  to  component 

tests  starting  for  the  modification  0 type  component;  i.e.,  uniform 

random  variates  were  drawn  in  sequence  and  indexed  as  U . . for  m = 0, 

m.j  , i 

1,  2 NTM;  j * 1,  2,  3,  •••  , NTF  ; and  i » 1,  2,  3.  start inq 

m 

w i th  the  UQ  j j tes  t . 

Each  uniform  variate  was  then  compared  with  the  appropriate 

underiving  modification  reliability  R to  determine  if  a success  or  a 

m 

failure  occurred  during  the  test.  The  number  of  tests  NT  and  the 

m.j 

number  of  failures  accumulated  NF  data  were  recorded  3t'ter  each  simu- 

m 

lated  test  such  that 
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NT  . = NT  . + 1 (NT  . incremented  by  one  test),  and  (3-57) 

m,j  m , j m,j 


NF  if  0 . . < R (success) 

m m , j , i m 


(3.58) 


NF  + 1 if  U . . > R (failure) 
m m, j , i = m 


for  m = 0,  1,2,  ...  , NTM;  j = 1,  2,  3,  •••  , NTF  ; and  i = 1,  2,  3, 

m 


...  If  the  test  was  a success,  then  the  simulation  proceeded  to  the  next 


test.  If  the  test  was  a failure,  then  the  number  of  failures  accumulated 


NF^  was  compared  to  the  total  number  of  failures  until  modification 


NTF  to  determine  if  a modification  was  appropriate.  If  NF  was  less 
m rr  r m 


than  NTF  , testing  continued  for  the  given  modification  and  number  of 
m 3 


tests  were  then  counted  until  the  next  failure;  i.e.,  NT  . , . If  NF 

m,j+l  m 


was  equal  to  NTF  , then  a modification  of  the  item  under  test  was  simu- 
m 


lated  and  testing  continued  for  the  next  modification;  i.e.,  NF  . and 

m+ 1 


| being  tallied  next. 


Testing  in  this  sequence  was  simulated  until  NF.,_U  was  equal 

NTM 


to  NTF  which  signalled  the  termination  of  a given  reliability  test- 


ing procedure  computer  simulation.  The  simulation  resulted  in  the  number 


of  tests  until  a failure  NT  . data  being  recorded  for  m = 0,  1,  2,  ...  , 

m,  j 


NTM  and  j = 1,  2,  3,  •••  , NTF  . These  data  constituted  the  total  data 

m 


gathered  from  a single  computer  simulation  of  the  discrete  model  reli- 


ability testing  procedure  for  a single  specified  underlying  modification 


reliability  progress  path;  i.e.,  a specified  set  of  reliabilities  which 


shall  henceforth  be  referred  to  as  a reliability  set.  The  reliability 


testing  procedure  was  then  replicated  one-hundred  times  (r  = I,  2,  3. 


NSIMS  * 1 00 ) such  that  reliability  testing  procedure  results  data  were 


actually  indexed  as  NT  . Since  ten  reliabilitv  sets  were  utilized 

m,j,r 


T* 


""T1 


■ 2F  rvrik  ITj**'*1 


v 


■ - --, 


for  evaluating  the  discrete  reliability  growth  model  with  three  different 

total  number  of  failures  until  modification  NTF  specified  for  each 

m 

reliability  set,  3000  reliability  testing  procedures  were  simulated  for 
the  discrete  reliability  growth  model  evaluation. 
k . Computer  Simulation  Data  Manipulation 

The  discrete  reliability  growth  model  of  equation  3-56  is  proposed 
as  a model  of  the  unknown,  underlying  modification  reliability  Rm  in- 
trinsic to  a component  as  it  proceeds  thru  a system  acquisition  cycle. 

It  is  aqainst  this  true  reliability  R that  the  model's  performance  was 

m 

measured. 

Equation  3-56  may  be  given  in  the  form 

R = 1 - exp  (-a  - 3m)  or  (3-59) 

m 

exp  (-a  - 3m)  = 1 - R (3-60) 

m 

for  m = 0,  1,2,  ...  , NTM.  The  logarithmic  transformation  of  equation 
3.60  yields 


(a  + 3m)  . = - 1 n ( 1 - R ) . 

J m J 


(3-61) 


for  m = 0,  1,  2,  ...  , NTM  and  j = 1,  2,  3,  •••  , NTF^  . 

An  unbiased  estimator  of  the  quantity  a + 3m  is  given  in 
reference  1 as 


0.0  i f NT  . = 1 (first  test  was  failure) 

m,  j 


(a  + 3m) 


m,  j 


(3-62) 


'*2*3*  » HT  . - I 

m,  j 


i f NT  . = 2 
m,  j 


for  m = 0,  1,2,  ...  , NTM  and  j ■ 1,  2,  3,  •••  , NTF 
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By  equation  3-6' 


- 1 n ( 1 -R  ) . * (a  + Sm) 

m m, j m,j 


(3-63) 


for  m - 0,  I,  2,  , NTM  and  j = 1,  2,  3 NTF^  . From  this 

relation  an  average  estimate  may  be  computed  as 


- 1 n ( 1 -R  ) = (a  + 3m)  = 

mm  m 


m NTF 


Z (a+3m)m,j-  (3<6M 


Letting 


Y = - 1 n ( 1 -R  ) , 

m mm 


then  enuation  3-61  may  be  written  as 


Y = ot  + 3 m 

mm  m 


(3-65) 


for  m = 0,  1,2,  ...  , NTM  which  is  of  the  form  Y = a + 3X.  Applying 

ordinary  least  squares  regression  estimates  as  given  in  reference  6,  the 

a and  3 estimates  are 
m m 


£ (i  - m)  Y. 

i =0 

m 

Z (i  ‘ ™)2 


(3-66) 


a = Y - 3 m 
m m 


(3-67) 
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It  . . 't  ..  .1  1 *x.  I . 


for  m - 1 , 2 , 3 , 


, NTM  where 


m+ 1 S Y i 


(3-68) 


m+l  S ' ' 


(3-69) 


Finally,  these  0s  and  fc>  estimates  are  utilized  in  the  discrete  re- 
in m 


liability  model,  equation  3-56,  to  produce  the  model  estimates  of  the 


unknown,  underlying  modification  reliability.  These  estimates  are  given 


(3-70) 


exp  ( 'o'  + p m) 

m m 


for  m » 1,  2,  3i  ...  . NTH.  Note  than  since  the  regression  procedure  re- 


quires a minimum  of  two  observations,  model  reliability  estimates  are  pro- 


duced from  the  first  modification  thru  the  last  modification  (NTM).  A 


reliability  estimate  for  original  version  of  the  component  is  derived 


from  equation  3-56  and  3-64  as 


(3.7D 


exp  [a  + tJm) 


Again,  since  the  reliability  testing  procedure  was  simulated  one- 


hundred  times  (r  ■ 1,  2,  3.  •••  . NSIMS  * 100)  for  each  reliability  set. 


one-hundred  estimates  of  the  modification  reliability  were  obtained  for 


each  modification  version  of  the  component  under  test.  The  mean  value. 


standard  deviation,  and  percentage  standard  error  of  these  estimates 


Ik 
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were  computed  as 


NS  I MS 


R - 


I 


m NS  I MS 


r V R' 

S i—j  m , r 


(3-72) 


r* ) 


NS  I MS 

S‘D‘^RV//nS  IMS-1  ^2 


(R  -'R'  )' 
m , r m 


and 


(3-73) 


P.S.E. 


S.°- 

m 


x 100 


(3-71*) 


for  m - 0 . 1,  2 NTM . 

To  examine  the  discrete  reliability  growth  model's  forecasting 
capability  a next  modification  reliability  forecast  was  made  for  modifi- 
cations m a 2,  3.  4,  •••  , NTM.  Forecasting  started  for  the  second 
modification  since  model  parameter  estimates  were  not  available  until 
testing  under  the  first  modification  was  completed.  No  forecast  was  made 
past  modification  NTM  because  there  would  be  no  underlying  modification 
reliability  with  which  to  compare.  Forecasts  were  made  as 


FR 


1 


m+ 1 


exp  (a  +3  (m+1)) 

m m 


(3.75) 


for  m * 2,  3.  4 NTM.  Finally,  forecast  modification  reliability 

statistics  similar  to  the  determined  modification  reliability  statistics 
in  eguations  3-72,  3-73.  and  3-74  were  computed  as 


FR 


m NS  I MS 


NS  I MS 


m , r 


(3.76) 
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'FR  / NS  I MS-  I 
m-v/ 


V (FR 
£—i  m , r 


FR  ) “■  , and 
m 


(3-77) 


P . S . E 

FR 


x 100 


(3-73) 


for  m ■*  2 , 3,  . •••  . NTM . 

To  evaluate  the  discrete  reliability  growth  model  ten  reliability 
sets  were  formed  for  use  as  tho  specified  underlying  reliability  progress 
paths.  These  reliability  sets  are  listed  in  table  3-27.  All  ten  reli- 
ability progress  paths  start  at  a relatively  low  reliability  of  0.200 
for  the  original  version  of  the  component  under  scrutiny;  i.e.,  RQ  * 0.200 
for  reliability  sets  1 thru  10.  The  paths  display  reliability  progress 
that  ranges  from  extremely  rapid  reliability  growth  (reliability  set  I; 

R.  ■ 0.200,  R,  3 0.925)  to  linear  growth  (reliability  set  6;  R , ■ R + 
u 1 m+ I m 

0.150)  to  permanently  stagnated  reliability  progress  (reliability  set  10; 
Rg  thru  R^  • 0.200).  Reliability  progress  was  simulated  only  for  compo- 
nents that  were  modified  a total  of  five  times  during  the  acquisition 
cycle;  i.e.,  NTH  » 5 and  m » 0,  I,  2,  3,  A,  5-  Each  reliability  set 
progress  path  was  simulated  for  three  different  total  number  of  failures 

until  modification  specifications;  i.e.,  NTF  » 1,  3.  and  5 failures 

m 

which  were  held  constant  from  modification  to  modification. 

5 . Model  Performance 

a.  Accuracy  Performance 

Figures  3-79  thru  3-98  present  all  the  cases  (ten  reliability 
sets)  of  the  discrete  reliability  growth  model's  capab i 1 i ty  to  determine 
and  forecast  the  unknown,  underlying  modification  reliability  progress 
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path  during  a reliability  testing  procedure  for  total  number  of  failures 

until  modification  NTF^  * 1 and  5-  These  graphs  display  the  "least  data" 

and  "most  data"  cases  which  can  be  contrasted  for  change  and  improvement 

as  the  testing  sample  is  increased.  For  the  NTF  » 3 cases,  the  accu- 

m 

racy  performance  fell  uniformly  between  the  NTF  - 1 and  NTF  = 5 accu- 

m m 

racy  performance. 

The  graphs  depict  the  specified  true  underlying  modification 
reliability  R^  progress  path  ( , solid  line),  the  mean  model  deter- 

mined modification  reliability  ^R^j  from  equation  3-72  for  each  modifica- 
tion version  of  the  component  under  test  (®  , circles),  and  the  mean 
model  forecast  modification  reliability  ”FR*m  from  equation  3-76  for  the 
second  thru  the  final  modification  version  of  the  component  (X  , crosses) 
plotted  versus  the  modification  number  m = 0,  1 , 2 , 3 , 4 . 5-  Note  that 
the  point  plotted  for  the  mean  mode)  determined  modification  reliability 
of  the  original  version  (m  = 0)  of  the  component  under  test  is  not  model 
determined;  rather,  it  is  the  mean  value  of  the  estimate  given  bv  the 
estimator  of  equation  3-71.  This  point  allows  the  accuracy  of  the  reli- 
ability estimator  utilized  to  be  examined  also. 

To  illustrate  those  quantities  graphed  in  the  accuracy  per- 
formance figures,  observe  on  figure  3-33  that  for  the  third  modification, 
the  specified  underlying  reliability  R^  was  0.90,  the  mean  model  deter- 
mined modification  reliability  ^R^  was  approximately  0.86,  and  the  mean 
model  forecast  modification  reliability  FR^  made  from  the  second  modi- 
fication was  approximately  0.77.  Since  this  graph  is  for  reliability 
set  3,  referring  to  table  3.27.  the  specified  modification  reliability 
for  the  third  modification  of  reliability  set  3 is  0-900  as  is  graphed 


on  figure  3-83.  Finally,  note  on  figure  3-83  for  the  unmodified  version 
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of  the  component  tested  (m  ■ 0)  that  the  reliability  estimator  of  equation 
3.71  produced  a mean  estimate  R^  of  approximately  0.175  for  the  true 
underlying  reliability  R^  of  0.200. 

With  the  exception  of  reliability  set  9 (figures  3-95  and 
3.96)  the  discrete  model  determines  the  shape  of  the  underlying  reliability 
progress  path  with  very  good  accuracy.  Forecasting  accuracy,  while  fairly 
good  for  the  "nice"  reliability  progress  path  sets  1 thru  4,  displays 
difficulty  with  the  anomalous  reliability  progress  paths  characterized 
in  reliability  sets  6 thru  10.  The  discrete  model  occasionally  produced 
negative  mean  forecast  estimates  of  reliability.  The  negative  mean  fore- 
cast reliability  estimates,  the  magnitude  of  which  never  exceeded  0.20, 
are  plotted  as  0.0  on  the  accuracy  performance  graphs.  For  example  see 
figure  3-97  for  mean  forecasts  at  modifications  2,  3,  and  4.  No  negative 
mean  forecasts  of  reliability  were  produced  when  the  total  number  of 

failures  until  modification  was  specified  as  five  (NTF  = 5);  i.e.,  the 

m 

"most  data"  case. 

The  discrete  model  accuracy  performance  graphs  display  the 
marked  improvement  that  takes  place  both  in  determining  and  forecasting 
reliability  status  as  the  specified  total  number  of  failures  until  modi- 
fication NTF  is  increased  from  one  to  five  which  essential lv  increases 
m 

the  test  data  which  the  model  can  utilize.  The  improvement  in  accuracy 
is  also  displayed  in  the  NTF^  = 3 accuracy  performance  graphs  which  are 
not  presented.  The  reliabilitv  estimator  of  equation  3-71  displays  the 
same  improvement  in  accuracy  when  the  testing  sample  size  is  increased 
as  may  be  seen  by  examining  the  graph  pairs  for  NTF^  » 1 and  NTF^  ■ 5 
at  the  modification  m » 0 point. 

Figures  3-79  thru  3-98  reveal  that  the  discrete  reliability 
model's  accuracy  performance  for  determining  the  underlying  reliability 
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is  in  general  slightly  pessimistic  in  the  instances  when  mean  estimates 
are  not  "on  the  money"  as  in  figures  3-3**  and  3-86.  Except  for  the  re- 
liability sets  1 and  2,  NTF^  = 5 cases  shown  in  figures  3 - 80  and  3-32, 
mean  model  forecasting  accuracy  tends  to  run  from  slightlv  pessimistic 
as  in  figure  3.90;  reliability  set  6,  NTF^  =*  5 to  grossly  pessimistic  as 
in  figure  3-91;  reliability  set  7,  NTF^  * ' or  figure  3-97;  reliability 
set  10,  NTF^  = 1.  The  general  trend  evidenced  by  the  graphs  is  that 

the  less  test  data  available  (NTF  =*  1)  the  more  pessimistic  the  mean 

m 

model  estimates,  determined  or  forecast.  The  same  is  true  of  the  reli- 
ability estimator  of  equation  3-71. 

Reliability  set  9 which  character i zes  a reliability  growth 
pattern  interrupted  by  a period  of  reliability  degradation  is  the  only 
set  with  which  the  discrete  model  experiences  a significant  problem. 

While  not  detecting  the  full  magnitude  of  the  degradation,  the  model's 
mean  determined  reliability  estimates  do  display  the  period  of  degrada- 
tion credibly.  See  figures  3-95  and  3-96.  However,  the  model's  mean 
forecast  modification  reliability  estimates  are  quite  contrary  to  the 
true  underlying  reliability  progress  path.  The  significant  problem 
being  that  the  period  of  degradation  is  not  forecast  until  the  true  under- 
lying reliability  status  has  recovered  from  the  period  of  degradation  and 
is  in  fact  displaying  significant  reliability  growth.  This  behavior  is 

apparent  even  for  the  "most  data"  case  NTF  = 5 in  figure  3.96.  While 

m 

model  forecasting  capability  suffers  in  the  "least  data"  case  for  reli- 
ability set  10,  NTF^  = 1,  the  discrete  model  copes  with  permanent  reli- 
ability stagnation  satisfactorily  as  seen  in  figures  3-97  and  3-98. 
b.  Variability  (Precision)  Performance 

Tables  3.28  thru  3-33  present  the  discrete  reliability  growth 
model's  variability  performance  for  determining  and  forecasting 
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the  reliability  status  of  a component  as  it  undergoes  modification  in  a 
system  acquisition  cycle.  Entries  in  the  tables  are  percentage  standard 
errors  as  computed  in  equations  3-74  and  3-78.  For  example  from  figure 
3.83  the  mean  model  determined  reliability  for  modification  3 from  reli- 
ability set  3 was  found  to  be  = 0.86  in  the  NTF^  = 1 case.  From 

table  3-28  the  percentage  standard  error  P.S.E.'T'  corresponding  to  this 

R3 

mean  model  determined  modification  reliability  is  20%.  Therefore,  by 
equation  3-7^  the  standard  deviation  for  the  observed  mean  model  deter- 
mi ned  reliability  is 

P-S-E-r' 

* 100  3 • °'86  *T§  • °-l7;  ■ (3-79) 

In  tables  3-28,  3-29,  and  3-30  the  discrete  model's  vari- 
ability performance  for  determining  modification  reliability  status  is 
displayed  for  the  specified  total  number  of  failures  until  modification 
cases  of  one,  three,  and  five  failures,  respectively.  Two  salient 
characteristics  displayed  by  the  resul  is  are  (1)  the  large  variability 
of  the  estimates  produced  by  the  reliability  estimator  of  equation  3-71 
and  (2)  the  large  variability  of  the  reliability  estimates  produced  by 
the  discrete  model  for  reliability  set  10,  the  permanently  stagnated 
reliability  case.  The  same  difficulty  is  displayed  to  a lesser  degree 
for  reliability  set  7 • 

If  reliability  set  10  is  not  considered,  then  a goal  of  4 0 % 

percentage  standard  error  is  achieved  bv  the  discrete  reliability  growth 

model  on  the  fifth  modification  for  the  NTF  = 1 and  3 cases  and  on  the 

m 

fourth  modification  for  the  NTF  = 5 cases.  If  reliability  set  7 is 

m 

also  dropped  from  cons iderat ion,  the  40«  goal  is  achieved  on  the  fifth 


Li 
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modification  for  NTF  * 1,  on  the  third  modification  for  NTF  = 3,  and 

m m 

on  the  second  modification  for  NTF  » 5*  Note  that  the  reliability 

m 

sets  characterizing  temporary  reliability  stagnation  and  degradation 

(sets  8 and  9)  did  not  have  to  be  taken  out  of  consideration  in  order  to 

find  variability  performance  at  a reasonable  level. 

Tables  3-31,  3-32,  and  3-33  present  the  discrete  model's 

variability  performance  in  forecasting  the  reliability  of  the  subsequent 

modification  version  of  the  component  under  test  for  the  NTF  =1,3,  and 

m 

5 cases.  Again,  the  worst  variability  performance  occurs  for  reliability 
sets  7 and  10.  Dropping  reliability  sets  7 and  10  from  consideration,  a 
b0%  percentage  standard  error  goal  is  achieved  only  in  the  case  of 
NTF^  * 5 at  the  fourth  modification.  Hence,  variability  performance  of 
the  discrete  reliability  growth  model  for  forecasting  is  not  particularly 
good. 
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IV.  CONTINUOUS  FAILURE  RATE  RELIABILITY 

grqwYh  models  comparison 

Codier  in  reference  2 gives  a derivation  in  which  an  equation  of  the 
form  of  the  continuous  instantaneous  failure  rate  reliability  growth 
model  (equation  3.37)  is  obtained  by  differentiating  the  equation  of  the 
continuous  cumulative  failure  rate  reliability  growth  model  (equation 
3.1)  with  respect  to  the  total  accumulated  test  time  TT  after  substitu- 
tion of  the  cumulative  failure  rate  estimator  of  equation  3. '3*  The 
derivation  essentially  proceeds  along  the  following  line: 


\ = — - £TT~a 

'TT  ' TT 


therefore , 


TF  - BTT 


(1-a) 


(‘♦.I) 

(4.2) 


Now , 


jjy  - (^TT('"a))  - (l-cOtJTT'0* 


and 


. JTF 

= TTT 


Hence  Codier  derives  an  equation 


(4.3) 

(4.4) 


\y  - ( 1 -a)  t$TT 


-a 


(4.5) 


and  claims  that  the  equation  is  the  continuous  instantaneous  failure 
rate  model 


\T  - ( 1 -a)  b TT  ; i .e. , 


v4.6) 
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equation  3-37.  The  implication  being  that  the  cumulative  model  parameters 
oc  and  jJ  are  equivalent  to  the  instantaneous  model  parameters  a and 
b , respectively;  and  so,  once  either  a and  £ or  a and  b have 
been  estimated,  it  is  unnecessary  to  estimate  the  other  parameter  pair. 

The  cumulative  and  i ns tantaneous  failure  rate  models  were  considered 
as  two  unrelated  models  which  were  proposed  to  model  different  aspects  of 
reliability  growth  from  the  viewpoint  of  this  thesis.  The  evaluation  of 
the  two  models  was  conducted  accordingly.  In  order  to  investigate  the 
proposition  that  the  cumulative  and  i nstantaneous  models  are  related  thru 
equivalence  of  model  parameters  a hybrid  cumulative  failure  rate  model  was 
devised  that  utilized  the  parameter  estimates  generated  for  the  instanta- 
neous failure  rate  model.  Also,  a hybrid  instantaneous  failure  rate 
model  was  devised  that  utilized  the  parameter  estimates  generated  for 
the  cumulative  failure  rate  model.  Specifically,  the  hybrid  cumulative 
failure  rate  model  is  given  by 


where  a.  and  b'.  were  determined  bv  the  i nstan  taneous  failure  rate 
i , r i , r 

model  equations  3.42  and  }. 43.  The  hybrid  instantaneous  failure  rate 
model  is  given  by 


■a. 


(1-a.  ) T.  TT. 

i , r i , r i , r 


' ,r 


(4.S) 


where  "'a!  and  ‘’J".  were  determined  bv  the  cumulative  failure  rate 
i , r i , r 

model  equations  3.21  and  3-22  after  appropriate  transformat  ions  given 
i n chapter  I I I .A. 4. 
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As  cumulative  and  instantaneous  failure  rate  model  parameter  esti- 
mates were  determined  for  each  phase  of  each  simulation  for  the  differ- 
ent lambda  sets  thev  were  applied  to  the  hvbrid  models  as  well  as  to 
their  appropriate  models.  Hvbrid  model  estimates  of  the  cumulative  and 
i ns  tan taneous  failure  rates  were  recorded  and  the  standard  statistics 
of  chapter  I I I were  compi led. 

Table  A. I contains  the  mean  cumulative  and  instantaneous  determined 
failure  rate  estimates  produced  bv  the  hybrid  models  for  the  lambda  set 
3,  ten  tests  per  phase  (NT.  ■ 10)  case  listed  with  the  appropriate  under- 
lying mean  test  time  weighted  average  cumulative  failure  rates 
(equation  3.30)  and  the  specified  underlying  i nstantaneous  failure  rates 
\.  (table  3*1).  Lambda  set  3 is  one  of  the  “nice"  failure  rate  sets 
for  which  both  the  cumulative  and  ins tantaneous  failure  rate  models  dis- 
played some  of  their  best  accuracy  and  variability  performance  ysee 
figures  3.5,  3.o,  3.7.  and  3.n-*;  and  tables  3.**  and  3.1&'.  Entries  of 

10.0000  for  the  hvbrid  cumulative  model  mean  determined  cumulative  failure 
rate  in  table  4.1  indicate  that  the  actual  mean  estimate  was  greater  than 
or  equal  to  10.0000. 

The  performance  of  the  hvbrid  .models  displayed  in  table  4.1  for  a 
"nice"  lambda  set  is  typical  of  the  behavior  of  the  hvbrid  models  for 
all  the  lambda  sets,  nice  or  anomalous.  The  hvbrid  cumulative  model 
( i nstantaneous  model's  estimated  parameters  substituted  into  the  cumula- 
tive model)  produced  failure  rate  estimates  of  magnitude  greater  than 

10.0  for  the  first  few  test  ohases  followed  bv  verv  erratic  estimates 
^including  negative  mean  estimates'!  which  displayed  no  relation  to  the 
nean  test  time  weighted  average  cumulative  failure  rates  supposedly  being 
modelled.  On  the  other  hand  the  hvbrid  instantaneous  node  I ^cumulative 
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TABLE  4. I 

HYBRID  CUMULATIVE  AND  INSTANTANEOUS  FAILURE  RATE  MODEL  SIMULATION  RESULTS 
LAMBDA  SET  3:  16  PMASES . 10  TESTS.  PHASE 


PHASE 
* **  ** 

MEAN  TEST 
TIME 

WEIGHTED 

AVERAGE 

CUMULATIVE 

FAILURE 

RATE 

********** 

MY  s>RIO 

CUMULATIVE 

MODEL  MEAN 

DETERMINED 

CUMULATIVE 

FAILURE 

RATE 

********** 

SPECIFIED 

INSTANTANEOUS 

FAILURE 

RATE 

************* 

MVSRI  D 

1 NSTANTANEOUS 
MODEL  MEAN 

determined 

INSTANTANEOUS 

failure 

RATE 

* ************ 

1 

0.7000 

0.6702 

0.7000 

0.6702 

4 % 

0.4703 

10.0000 

0.3530 

0.3605 

3 

0.3565 

10.0000 

0.2330 

0.3043 

A 

0 . 2363 

10.0000 

0.1300 

0.2669 

5 

0.2392 

10.0000 

0 . 1 4*0 

0.2362 

6 

0.2064 

-0.1003 

0. 1220 

0.2173 

0. 1 3 1 3 

1.9321 

0. 1060 

0.2016 

O 

0. 1625 

-0.0171 

0.0933 

0.133a 

A 

0. 1474 

-0 . 0065 

0.0337 

0.1 ’96 

10 

0 . 1 346 

0.0532 

O.Q’60 

0 . 1 634 

1 1 

0. 1241 

-0.020c 

0.069’ 

0. 1626 

i : 

0.1152 

0.0237 

0 . 0c4* 

0. 1R4$ 

13 

0. 1076 

0.0070 

0.0600 

0.1497 

u 

0. 1009 

0.2491 

0.0562 

0.1442 

15 

0.0952 

0.0513 

0.0529 

0.1402 

16 

0.0900 

0 . 0060 

0.0500 

0.135’ 
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model's  estimated  parameters  substituted  into  the  instantaneous  model) 
produced  what  appeared  to  be  a good  failure  rate  estimate  for  the  first 
one  or  two  phases  (i  * 2,  3;  models  do  not  estimate  for  the  first  phase) 
but  quickly  transitioned  to  a dampened,  sedate  pattern  of  estimates  which 
was  very  unresponse i ve  to  even  the  most  radical  variations  in  the  under- 
lying instantaneous  failure  rate  progress  path.  Also,  the  hybrid  in- 
stantaneous model's  failure  rate  estimates  after  the  second  or  third 
phase  generally  exhibited  increasing  magnitude  error  except  for  crossing 
situations  in  the  cases  of  temporary  stagnation  or  degradation  of  the 
snecified  underlying  i ns  tan taneous  failure  rates. 

Regarding  the  negative  failure  rate  estimates  produced  by  the  hybrid 
cumulative  model  (instantaneous  model  parameters  in  cumulative  model), 


the  following  observations  on  the  oarameter  estimates  aN  arid 

i , r 


* . r 


of  the  continuous  instantaneous  failure  rate  model  were  made  during  the 
reliability  testing  procedure  computer  simulations.  During  the  simula- 
tions the  magnitude  of  the  parameter  estimate  ^a*  was  continually  very 

* * 

close  to  1.0  often  being  more  or  less  than  1.0  only  in  the  tenth, 
eleventh,  or  twelfth  decimal  place.  From  the  i ns tantaneous  model  equa- 
tion 4.6  it  can  be  seen  that  logically  if  the  <*a')‘  estimate  is  less 

i , r 

than  1.0,  the  ^ estimate  should  be  positive;  and,  if  the  esti- 

i ,r  r i ,r 

mate  is  greater  than  1.0,  the  o?  estimate  should  be  negative  in  order 

* I ^ 

to  produce  positive  estimates  of  the  instantaneous  failure  rate.  Exactly 
such  behavior  was  observed  during  the  computer  simulations;  as  the 
magnitude  of  the  'a'j‘  estimate  moved  to  either  side  of  1.0  the  sign 


of  the  b.  r estimate  would  "flip-flop"  accordinglv.  Of  course  if  a 


negative  b.  r estimate  is  plugged  into  the  cumulative  failure  rate  model 
for  8 in  equation  4.1,  a negative  estimate  of  the  cumulative  failure 
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rate  necessarily  results  no  matter  what  the  sign  or  magnitude  of  the 

”a^  estimate  which  is  substituted  for  a (save  a value  of"a>  =*  °°) . 
i , r i ,r 

Hence,  it  is  very  plausible  for  the  hybrid  cumulative  model  to  have  pro- 
duced negative  estimates  of  cumulative  failure  rate. 

In  contrast  to  the  hybrid  cumulative  model's  erratic  failure  rate 
estimates  the  hybrid  i ns  tan taneous  model  (cumulative  model  parameters 
in  instantaneous  model)  produced  failure  rate  estimates  which,  after  one 
or  two  test  phases,  displayed  a "sluggish"  growth  pattern  and  generally 
increased  in  magnitude  error  from  the  underlying  instantaneous  failure 
rate  A.  being  modelled.  This  situation  is  not  surprising  because  the 
cumulative  model  parameters  being  employed  in  the  hybrid  instantaneous 
model  are  parameters  utilized  in  modelling  a "smoothed"  quantity,  the 
underlying  cumulative  (average)  failure  rate  that  is  collectively  charac- 
teristic of  all  the  versions  of  an  item  tested  thru  a given  point  in  the 
acquisition  cycle.  As  the  history  of  the  item  stretches  farther  and 
farther  (total  accumulated  test  time  TT  increases)  the  cumulative 
failure  rate  of  the  item  becomes  increasingly  insensitive  to  the  current 
i ns  tan taneous  failure  rate  intrinsic  to  the  item.  Hence,  the  parameters 
of  the  cumulative  model  likewise  become  increasingly  insensitive  to  the 
current  instantaneous  reliability  status.  When  these  relatively  stable 
parameters  are  utilized  in  another  model  whatever  its  form,  "smoothed" 
stable  output  is  reasonably  expected. 
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V.  EVALUATION  SUMMARY  AND  CONCLUSIONS 


This  thesis  examined  three  reliability  growth  models  for  which  accu- 
racy , precision,  and  robustness  character i s t i cs  were  investigated  over  a 
wide  variety  of  true  underlying  growth  patterns,  both  "nice"  and  anomalous. 
The  evaluation  of  a continuous  cumulative  failure  rate  model,  an  instanta- 
neous failure  rate  model,  and  discrete  reliability  model  was  accomplished 
by  computer  simulation  of  reliability  testing  procedures  that  were  appro- 
priate to  the  reliability  growth  model  tvpes  for  a systems  acquisition 
cvcle.  Performance  of  the  models  both  in  tracking  and  predicting  true 
underlying  reliability  progress  patterns  was  measured  for  mean  accuracy 
and  variability  (precision)  over  progress  patterns  which  included  most 
situations  that  can  be  encountered,  both  good  and  bad.  The  models'  per- 
formance characteristics  are  summarized  in  the  following  paragraphs. 

A.  CONTINUOUS  CUMULATIVE  FAILURE  RATE  RELIABILITY  GROWTH  MODEL 

The  continuous  cumulative  failure  rate  model  examined  general Iv  dis- 
played very  good  to  excellent  accuracy  performance  in  tracking  both  "nice" 
and  anomalous  true  underlying  cumulative  failure  rate  patterns.  The 
cumulative  model  did  exhibit  difficulty  in  determining  and  forecasting 
true  underlying  patterns  that  characterize  rapid  reliability  growth 
(failure  rate  decay)  in  the  latter  phases  of  the  systems  acquisition 
cycle.  The  difficulty  was  evidenced  bv  mean  model  determi ned,  forecast 
cumulative  failure  rate  estimates  that  diverged  from  the  true  underlying 
values  on  the  pessimistic  side;  thus,  the  cumulative  model  provided  esti- 
mates that  could  he  considered  as  upper  bounds  on  failure  rate  in  this 
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situation.  The  other  anomalous  situation  with  which  the  cumulative  model 
experienced  difficulty  in  tracking  and  predicting  accuracy  was  one  of 
reliability  growth  interrupted  by  a period  of  reliability  degradation. 

The  modelled  cumulative  failure  rate  path  both  lagged  the  true  underlying 
cumulative  failure  rate  path  and  failed  to  reflect  the  full  magnitude  of 
the  failure  rate  degradation.  Thus  dependence  on  the  model  in  this  situ- 
ation could  hamper  response  to  the  degradation  bv  giving  a "too  little 
too  late"  signal  of  the  problem.  The  cumulative  model  coped  with  true 
underlying  cumulative  failure  rate  patterns  that  were  permanently  stag- 
nated quite  adequately. 

As  the  reliability  testing  procedures  of  the  acquisition  cycles  com- 
pleted more  test  phases,  the  cumulative  model's  variability  (precision) 
performance  improved  uniformly  for  both  "nice"  and  anomalous  true  under- 
lying failure  rate  paths.  Variability  percentage  standard  error  goals 
were  satisfied  earlier  and  earlier  in  the  acquisition  cycle  as  more 
testing  data  was  made  available  to  the  model.  This  good  variabilitv 
performance  provides  a degree  of  confidence  in  employing  the  cumulative 
failure  rate  model  in  actual  systems  acquisition  programs  where  the 
intrinsic  cumulative  failure  rate  status  is  trulv  unknown. 

8.  CONTINUOUS  INSTANTANEOUS  RA I LURE  RATE  RELIABILITY  GROWTH  MODEL 

The  continuous  i ns tantaneous  failure  rate  model  examined,  while 
exhibiting  the  capability  to  track  and  predict  the  shape  of  various 
unknown,  underlying  i ns  tan taneous  failure  rate  path  types  with  very  good 
accuracy  and  robustness,  consistently  produced  opt i mi s t i ca I 1 v biased 
estimates  of  the  instantaneous  failure  rate.  If  hi  >s  must  be  accepted 
in  a reliability  growth  model,  then  pessimistic  bias  is  preferred  to 
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optimistic  bias.  Consistent  pessimistic  bias  permits  managers  to  utiiiie 
the  model  produced  estimates  as  lower  bound  type  values  and  have  confi- 
dence that  reliability  status  is  being  observed  from  the  "right  side  of 
the  fence".  On  the  other  hand  consistentlv  optimistic  model  produced 
estimates  impart  a great  deal  of  uncertainty  as  to  how  bad  reliability 
status  might  be  and  what  degree  of  corrective  action  Is  reguired. 

The  anomalous  situations  in  which  the  instantaneous  model  experienced 
difficulty  in  tracking  and  predicting  the  shape  of  the  true  failure  rate 
path  were  (1)  the  initial  period  of  recovery  in  reliability  growth  after 
a period  of  temporary  reliability  stagnation,  (2)  permanently  stagnated 
reliability  growth  at  high  failure  rates,  and  (3)  reliability  growth  in- 
terrupted by  a temporary  period  of  reliability  degradation.  In  the  case 
of  temporarv  reliability  degradation  the  i nstantaneous  model  charted  a 
determined  failure  rate  path  that  was  an  exaggeration  of  the  underlying 
instantaneous  failure  rate  path;  i.o.,  the  periods  of  growth  were  displayed 
opt imi s t i ca I I s and  the  period  of  degradation  was  portrayed  to  a magnitude 
greater  than  it  was  in  truth.  The  instantaneous  model's  forecasts,  while 
accurately  capturing  the  shape  of  the  underlying  progress  path,  were  con- 
sistently optimistic  during  the  period  of  degradation. 

The  instantaneous  model's  accuracy  performance  was  generally  better 
for  contracted  acguisition  cvcles  and  suffered  when  the  reliability  test- 
ing procedure  was  extended  to  encompass  a large  number  of  test  phases. 
Because  of  the  model's  consistent  optimistic  bias,  accuracy  performance 
improved  as  the  true  underlying  failure  rate  decreased  thereby  "sandwich- 
ing" the  mode!  estimates  between  the  true  failure  rate  and  0.0.  Also, 
the  instantaneous  model  routinely  furnished  "of f-the-scale"  forecasts  for 
the  first  one  or  two  phases  of  an  acquisition  cvcle. 

The  instantaneous  model's  demonstrated  capab i 1 i tv  to  determine  and 
forecast  the  true  underlying  reliability  status  was  overshadowed  by  its 
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poor  variability  performance  which  (I)  never  achieved  a really  comfort- 
able percentage  standard  error  goal  uniformly  for  all  failure  rate  pat- 
terns simulated  and,  more  damaging,  (2)  oscillated  as  test  phases  of  the 
reliability  testing  procedure  were  accomplished.  Because  of  this  poor 
variability  performance,  the  i nstantaneous  failure  rate  model  cannot  be 
employed  by  itself  with  any  degree  of  confidence. 

C.  DISCRETE  RELIABILITY  GROWTH  MODEL 

The  discrete  reliability  growth  model  examined  generally  displayed 
very  good  determined  reliability  accuracy  performance  even  for  the  most 
restricted  test  data  cases.  Slight  pessimistic  bias  was  demons t rated  for 
model  determined  reliability  status  while  a greater  magnitude  of  pessi- 
mistic bias  was  present  in  model  forecast  re  I i ab i I i ty . Again,  pessimis- 
tic bias  is  favored  over  optimistic  bias  in  charting  a reliaoility  prog- 
ress path.  The  degree  of  pessimism  in  determined  and  forecast  re  1 i ab i 1 i ty 
decreased  markedly  as  more  test  data  on  each  modification  version  of  an 
item  were  gathered.  In  cases  of  limited  test  data  the  discrete  model 
often  forecast  negative  estimates  of  reliability  for  the  early  modifi- 
cation versions  of  a component  under  test. 

For  the  anomalous  situation  of  temporary  reliability  degradation  the 
discrete  model  (l)  failed  to  determine  the  full  magnitude  of  the  degrada- 
tion and  (2)  provided  reliability  status  predictions  that  lagged  the 
true  underlying  reliability  path  outcomes  significantly.  In  actual  test- 
ing this  performance  characteristic  would  give  a delayed  signal  of  a 
situation  that  required  corrective  action.  Also,  gathering  additional 
test  data  failed  to  remedy  this  deficiency  in  the  discrete  model's  per- 
formance . 
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Variability  performance  of  the  discrete  model,  while  generally  good, 
revealed  difficulty  with  the  permanently  stagnated  reliability  case  and 
the  case  of  rapidly  increasing  reliability  during  the  latter  modifica- 
tion versions  of  an  item.  Variability  performance  improved  uniformly  for 
all  progress  path  characterizations  as  more  mod i f i cat  ions  of  the  item  be- 
ing tested  were  accomplished.  Also,  variability  goals  were  satisfied 
earlier  in  the  acquisition  cycle  as  more  complete  testing  was  accomplished 
on  each  modification  version  of  the  item.  This  nice  behavior  of  the  dis- 
crete model  lends  confidence  to  its  utilization;  and  therefore,  the  dis- 
crete model  is  preferred  to  the  continuous  i ns tantaneous  failure  rate 
model  for  obtaining  a measure  of  the  current  or  " i ns  tan taneous"  reli- 
ability status  of  an  item  proceeding  thru  an  acquisition  cycle. 

0.  GENERAL  OBSERVATIONS 

For  determining  current  reliability  status  it  may  be  suggested  that 
rather  than  employing  the  reliability  growth  models,  why  not  utilize  the 
point  estimators  of  reliability  status  which  were  used  to  provide  data 
to  the  models?  Although  the  performance  of  the  point  estimators  appro- 
priate to  each  reliability  growth  model  were  only  observed  at  one  point 
for  each  model  in  the  reliability  testing  procedure  simulations  (first 
phase  of  testing  or  mod  0 version  of  a component),  at  that  point  the 
estimators  displayed  very  poor  variability  performance  upon  which  the 
reliability  growth  models  improved  rapidly  and  significantly  which  tends 
to  negate  any  confidence  in  utilizing  the  point  estimators  based  on  their 
good  accuracv  performance. 

In  retrospect  the  performance  characteristics  demonstrated  bv  the 
reliability  growth  models  examined  suggest  that  as  a model  is  emp loved 
the  estimated  reliability  progress  path  it  produces  be  compared  with  the 
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appropriate  model  accuracy  performance  graphs  in  chapter  III.  If  the 
estimated  reliability  progress  path  corresponds  to  a case  for  which  the 
computer  simulation  performance  results  show  the  model's  performance 
was  good,  then  confidence  can  be  placed  in  the  estimated  progress  path. 

On  the  other  hand  if  the  estimated  path  corresponds  to  one  of  the 
anomalous  cases  where  simulation  results  revealed  a deficiency  in  the 
model's  performance,  then  the  simulation  performance  results  at  least 
give  an  indication  of  which  direction  the  true  underlving  reliability 
progress  path  lies. 

Finally,  although  use  of  the  continuous  i ns  tan taneous  failure  rate 
reliability  growth  model  is  gues t ionab le , since  the  test  data  re  i red 
for  the  i ns  tan taneous  model  is  identical  to  the  data  collected  for  the 
continuous  cumulative  failure  rate  model,  application  of  the  instanta- 
neous model  simultaneously  with  the  cumulative  mode  I utilizing  the  method 
described  in  the  preceding  paragraph  may  provide  some  insight  to  the 
shape  of  the  true  underlying  i ns tantdneous  failure  progress  path  from 
the  instantaneous  model.  8ut  the  comparison  oerformed  between  the  two 
continuous  failure  rate  models  definitely  indicates  that  the  cumulative 
failure  rate  and  i nstantaneous  failure  rate  models'  parameters  not  be 
interchanged  based  on  their  hypothetical  egui valence. 
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APPENDIX  A 


Derivation  of  Ordinary  Least  Squares  Regression  Estimates  of  the 
Continuous  I ns  tan taneous  Failure  Rate  Reliability  Growth  Model  Parameters 


The  continuous  instantaneous  failure  rate  model  reliability  growth 
model  as  given  in  equation  3.37  may  be  written  as 


X.  - b ( I - a ) TT . d . 


Taking  the  logarithmic  transformation  of  this  equation  yields 


In  X.  » In  b(l-a)  - a In  TT. 


(A.I) 


(A. 2) 


E-E 


Y.  =■  In  X.  , 

I I 


X.  - In  TT.  , 

i i 


, and 


There  fore , 


Y.  - In  b(l-a)  - aX. 


(A. 3) 


The  residual  error  for  regression  is 


C.  - Y.  - Y.  - Y.  - [In  l-a)  JX. ] . 
iiii  i 4 


(A.  M 


The  goal  of  Che  regression  is  to  minimize  the  sum  ot  the  residual  errors 
sguared;  i .e. , 

mi"  T,~\2  " Z(Yi'^i):"  E[Vi*ln  M,-J>  ^3Xi]:- 

(A. 5) 

Taking  partial  derivatives  of  eguation  A. 5 with  respect  to  b and  a ; 
then  eguating  the  partials  to  0 yields  the  following  two  eguations  in 
two  unknowns: 


(0  a 


I>Y 


>:)  3 V , 


Y"  t 2 IY ; "In  b(l-j)  * jx.l(-  ^ (I-,)]  ■ 0 

V*  [2  tV.  -In  * nX.M-  — ] r(-t>I+X . ) ] - 0 

i i b ( I -3 ) i 


S i mp I j f [ c3  C i on  of  (I)  Jnd  (2)  yields  in  sequence: 


[Y . - In  bil-a)  a\.]  - 0 . and 
[ (Y  . * I n b ^ I - a)  + aX  . ) V7y|~T  + X . ) ] -0  . Then , 


(I")  - ^ln  b(l-a) 


aX.  , 


ErnTT  Y * EVi  ■ E rrrrr  - 


^JaX . “ . Finally, 
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Substituting  and  clearing  terms  in  (2*)  leaves 


Solving  for  a in  (2--*)  gives 


(2***)  a(X  £ X.  - £x.2) 


(A. 7) 
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I 


| 
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